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SYSTEM FOR NON-DISRUPTIVE INSERTION AND 



REMOVAL OF NODES IN AN ATM SONET RING 



FIELD OF INVENTION 



The invention relates generally to asynchronous transfer mode (ATM) rings and, 
more particularly, to mechanisms for insertion and removal of ring nodes. 



Communication rings include a plurality of nodes that are interconnected by a 
communication medium, such as fiber optic cable. In known prior ATM rings, the traffic 
on the ring is disrupted when a node is inserted into the ring, that is, when the node is 
brought into the ring operations, and/or when a node is removed from the ring operations. 
In order for the newly inserted node to operate in the ring, the existing nodes must be 
configured to make use of the virtual paths associated with the new node. In one known 
ATM ring, the nodes on the ring must be re-configured individually and manually, that is, 
each must be reprogrammed. Accordingly, the traffic on the ring is disrupted as the 
operations of each of the respective ring nodes are interrupted so that the node can be 
updated to make use of the new virtual paths. 

The traffic on the ring is also disrupted each time a ring node has a catastrophic 
failure and must be removed from the ring operations. When a node fails, the functioning 
ring nodes must again be reconfigured. In such systems, the traffic on the ring is also 
further disrupted when the failed node is re-booted and re-inserted into the operations of 
the ring. Ring traffic may also be disrupted when a node is temporarily removed from 
the ring for service or upgrading and thereafter re-booted and re-inserted. 

The reconfigtiration of the nodes in the prior system must be performed under the 
control of the system manager. Accordingly, the time to reconfigure each node, and thus, 
the time during which traffic on the ring is disrupted at each node, and collectively over 
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the ring, is relatively long. Further, the time it takes to insert a node increases as the 
number of nodes on the ring increases. 

As the rings become larger and more traffic is sent over them, the longer 
disruptions in the traffic are not as easily tolerated. This is particularly the case when the 
5 ring is shared by multiple customers, and any disruption adversely affects more than the 
customer that is, for example, joinmg the ring. Accordingly, what is needed is a system 
for inserting and removing a node without disrupting the traffic on the ATM ring. 

SUMMARY OF THE INVENTION 

The invention is a system and a method of operating the system for non- 
10 disruptively inserting a node into the operations of an ATM ring by first including the 
node as a virtual path pass through and then using the ring operations to update the 
routing tables at the new node and the existing nodes to include therein the one or more 
virtual paths associated with the new node. The system non-disruptively removes a node 
fi-om the ring essentially by reversing the insertion operations. The system and method 
15 are described below in terms of a SONET ring that handles ATM traffic, but may be used 
in any ATM ring. 

Basically, the node to be inserted establishes conununications with a ring hub 
node over an established intra-ring management channel. The node and the ring hub 
node then exchange information over the intra-ring management channel and later over a 

20 virtual path that is assigned to that node, to bring the new node mto the ring operations in 
stages without disrupting the flow of ATM traffic over the existing virtual path 
connections. For ease of understanding, we refer hereinafter to the node that is inserted 
into the ring as the "new node." 

More specifically, once the new node is physically connected into the ring it 

25 operates as an optical bypass. The new node then performs its initializing routines and, 
as necessary, conmiunicates with the hub node, to allow the traffic on the existing virtual 
paths to pass through the processor on the new node. The new node next requests that the 
hub node assign one or more virtual paths for directing traffic between the new node and 
the other nodes on the network, that is, between the new node and the other ring nodes 

30 and also the network nodes that are external to the ring. In response to the request, the 
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hub node assigns the virtual path or paths to the new node and establishes a signaling 
channel to the node over one of the virtual paths. The hub node also notifies the other 
nodes of the assignment, and downloads to the new node routing tables that include the 
established connections over the existing virtual paths. 

The new node is thus brought fully up-to-date. Accordingly, it may participate in 
traffic shaping on existing virtual circuits, and in call set up and call tear dovm operations 
over its assigned virtual paths, as discussed in more detail below. 

When a node fails, the system reverses the steps discussed above to remove the 
node from the ring without further disruption of the traffic on the ring. The hub node 
leams of the node failure when the failed node no longer commxmicates with the hub 
node, as discussed in more detail below. The hub node then instructs the operative nodes 
to tear down the virtual paths and associated virtual circuits that originate or end at the 
failed node. The nodes continue to maintain the other virtual path and virtual circuit 
connections over the ring, and the ATM traffic that is directed to the operative nodes is 
thus not disrupted. When the failed node is later re-booted, it is brought back into the 
ring operations without disrupting the traffic on the ring using the steps discussed above. 

The operations of the current system are in contrast to known prior systems in 
which the insertion of a new node and/or the failure and re-insertion of a previously 
operative node requires system-operator controlled reconfiguration of the nodes on the 
ring. 



BRIEF DESCRIPTION OF THE DRAWINGS 

The invention description below refers to the accompanying drav^ngs, of which: 
Fig. 1 is a functional block diagram of a system constructed in accordance with 
the invention; 

Fig. 2 is the system of Fig. 1 with an additional ring node; 
Fig. 3 is a flow chart of the operations of the system of Figs. 1 and 2 to insert a 
node; and 
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Fig. 4 is a flowchart of the operations of the system of Figs. 1 and 2 to removing a - 
node.. 

DETAILED DESCRIPTION OF AN ILLUSTRATIVE EMBODIMENT 

Referring now to Fig. 1, a system 10 that may be part of an ATM network 
includes a plurality of nodes 12i, 122, ... 12i that are referred to collectively as nodes 12. 
One of the nodes, for example, node 12i, operates as a ring hub node to control ring 
operations. 

The nodes 12 are interconnected by a primary ring 14 and a secondary ring 16. 
The exemplary system 10 is a unidirectional SONET ring, with information in the form 
of ATM cells and frames transmitted in both directions over the primary and secondary 
rings. The system 10 may, however, be any type of ATM ring. We discuss below how a 
new node is non-disruptively inserted into the ring and how a failed node is non- 
disruptively removed from the ring. First, however, we discuss in general the operations 
of the system 10. 

1. Ring Operations In General 

Each node 12 receives information over the rings 14 and 16 through incoming 
ports 20 and sends information over the rings through destination ports 22. The nodes 12 
also connect to other systems (not shown), including other networks or rings, or to 
various user terminals, through other incoming and destination ports 24 and 26. 

The ATM cells are sent over the rings from source nodes to destination nodes in 
virtual circuits. The system groups the virtual circuits into virtual paths that include other 
virtual circuits with the same source and destination nodes. 

Each virtual path is identified by a virtual path index (VPI), and each virtual 
circuit is similarly identified by a virtual circuit index (VCI). The VPI and VCI values 
are included in ATM cell headers, and are used by successive nodes to determine how to 
route the cells. 

Intermediate nodes manage the cells on a virtual path basis. When an 
intermediate node receives cells over a virtual path on the primary ring, the node 
forwards the cells to succeeding nodes on the primary ring over the same virtual path. 
The intermediate node may, however, use the virtual circuit information for traffic 
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shaping in the virtual path, in a conventional manner. Similarly, when an intermediate 
node receives cells over a virtual path on the secondary ring the node forwards the cells 
to succeeding nodes on the secondary ring over the same virtual path, and uses the virtual 
circuit information for traffic shaping. 

Each destination node maintains routing tables (not shown) that reference at least 
the associated virtual circuits over both the primary and secondary rings and the 
destination ports to which the virtual circuits are directed. The destination nodes select 
one ring interface, based on the quality of the traffic received over both rings, and direct 
the traffic fi-om the selected interface to the appropriate ports. 

The nodes 12 exchange ring management information with the hub node 12\ over 
a dedicated intra-ring management channel. In particular, the hub node 12 \ and the other 
ring nodes 12 exchange ring topology information over the intra-ring management 
channel. In the exemplary system 10, the topology information includes updates to 
virtual path and virtual circuit routing tables (not shown) that are maintained at each of 
the nodes, and also information relating to the responsiveness or non-responsiveness of 
the various nodes. The ring nodes 12 also provide to the hub node 12i certain other 
information concerning the integrity of the primary and secondary rings by, for example, 
periodically sending 0AM cells to the hub node over the intra-ring management channel. 

2. Ring Operations For The Insertion Of A Node 

Referring now also to Fig. 2, before a new nodel2j is inserted into the system 10, 
that is, before the node participates in ring operations, the cabling and the node hardware 
are installed in a conventional manner. Accordingly, to install a new node 12j between, 
for example, the nodes I23 and I24, the cabling 15 for the primary ring 14 and the cabling 
17 for the secondary ring 16 that connect nodes I23 and I24 are unplugged from the ports 
20 and 22 of the node I24. While the cabling is unplugged, the traffic between the nodes 
123 and 124 is re-routed aroimd the ring. The unplugged cabling is then plugged into the 
appropriate incoming and destination ports of the new node 12j and additional cabling is 
run between the new node and the node I24. Electrical connections for the two rings are 
thus made through the connection hardware 13j of the new node 12j. The added cabling 
is depicted in the drawing by dotted lines. 
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Referring as well to Fig. 3, once the cabling is in place, tEe new node 12j operates 
as an optical bypass, to physically pass ring traffic between the nodes I23 and I24 (step 
300). The new node 12j must then be inserted into the ring operations, so that users 
connected to the ring through the node 12j can send and receive information over the 
ring. 

The new node 12j performs its initialization routines, places all virtual paths in 
pass through mode, and communicates v^th the hub node 12i over the intra-ring 
management channel to exchange information that allows a node processor in the new 
node 12j to control traffic passing through the new node. In particular, the hub node and 
the new node 12j exchange information that allows the node processor to process the 
ATM traffic on the existing assigned virtual paths through the ports 20j and 22j (steps 
302, 304, 306). In the example, the new node contains a list of a default range of virtual 
paths from which all of the virtual paths on the ring are assigned, and the new node 
essentially configures itself, without hub node control, to operate as a virtual path pass 
through. If the new node did not contain such a list, the hub node would instead provide 
a list of the assigned virtual paths. 

The new node 12j next sends a request to the hub node 12i over the intra-ring 
management channel, asking the hub node to assign to the new node one or more virtual 
paths. The new node will later use these virtual paths to establish connections over 
virtual circuits to the other nodes in the system, both on and external to the ring (step 
308). With the new node requesting its virtual path assignment, the hub node is freed 
from monitoring the passive activities of this node and the other new nodes that are being 
brought into the ring operations. Accordingly, the hub node may instead manage the 
dynamic operations on the ring, such as call set up and tear down. 

In response to the request from the new node, the hub node assigns one or more 
virtual paths to the node. The hub node may then notify the other nodes of the new 
assignment (step 310). Alternatively, if the path assignment is v^thin the default range, 
the hub node may instead update the topology information to include the new node and 
rely on call set up operations to update the routing tables when connections are 
established over the newly assigned virtual paths, as discussed below. The ring nodes 12 
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may instead learn of the new node through cells that are periodically introduced to test 
ring continuity and/or to verify ring topology. 

The hub node next establishes a signaling channel over one of the virtual paths 
assigned to the new node. The hub node and the new node can then exchange call set up 
and tear down information over the signaling channel (step 312). The hub node next 
downloads to the new node 12j over the management channel a set of up-to-date routing 
tables that include all of the existing virtual path assignments and/or the associated virtual 
circuit connections (step 314). The routing tables are preferably the tables discussed in 
co-pending United States patent application Serial No. 09/344,845 entitled RAPID CALL 
ESTABLISHMENT IN ATM RINGS which is assigned to a common assignee and is 
incorporated herein by reference. In the exemplary system, the hub node also downloads 
an error checking code, such as a check sum, that is used by the new node 12j to verify 
that the routing tables were downloaded without error. 

The node 12j may now fully participate in the ring operations. More specifically, 
the node 12j sends over the established signaling channel requests to the hub node 12i to 
set up calls from associated users as virtual circuits over a newly assigned virtual path, 
and later to tear down inactive connections over the assigned virtual path. This call set 
up and tear down information is also sent by the hub node to the other nodes on the ring, 
and these nodes then update their routing tables to include the connections. Based on 
instructions and update information received by the new node at various times from the 
hub node, the node 12j updates its routing tables as new connections are established 
between the various other ring nodes, and as various connections are tom down. Also, 
the new node uses the virtual circuit information in the routing tables to shape traffic over 
the various virtual paths in a conventional manner. 

3. Ring Operations For Removal Of A Failed Node 

Referring now to Fig. 4, when a node 12 fails, the system 10 removes the failed 
node from the ring operations, without disrupting the operations of the remaining ring 
nodes, essentially by reversing the steps for the insertion of a node. The hub node learns 
of the failure of, for example, node I23, when the node I23 stops communicating with the 
hub node over the signaling channel established between them. The hub node may also 
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learn of the failure of the node I23 when the hub node does not^^uring a predetermined 



maximum time interval, receive 0AM cells or other cells that are periodically originated 
by the node I23. Alternatively, the hub node may leam of the failure v^hen the node does 
not respond appropriately to the 0AM cells or other test cells that the hub node originates 
and sends over the ring to test ring continuity and/or to verify ring topology (step 400). 

Once the hub node has determined that the node I23 has failed, the hub node 
instructs the other nodes on the ring to, as necessary, re-route traffic through the operative 
nodes as part of a protection switching operation (step 402). If the electrical connections, 
that is, the cables and associated hardware, through the failed node are functional, the 
failed node I23 operates by default as an optical by-pass and the ring traffic need not be 
re-routed. 

The hub next instructs the operative ring nodes to tear dovm the virtual circuits 
that originate from or are directed to the failed node (step 404). In response, the nodes 
update their routing tables. The hub node then sends updated ring topology information 
to the remaining nodes, and instructs them to remove from their routing tables the virtual 
paths associated v^th the failed node. At the same time, the remaining ring nodes 12 
continue to send and receive ATM traffic associated with the operative ring nodes over 
the ring in the usual manner, that is, over the existing virtual paths and associated virtual 
circuits. 

When the failed node 12 is later replaced or re-booted, the new or newly operative 
node is inserted into the ring operations in accordance with the steps discussed above 
v^th reference to Fig. 3, and the traffic over the ring is not disrupted. The same node 
insertion operations may also be followed aflter a node has been taken temporarily out of 
ring operations for upgrading and/or service. 

The system and method of operating the system described above allow ring traffic 
to travel the ring without disruption while nodes are added or failed nodes are removed 
from the ring. This is in contrast to the operations of known prior ATM rings that require 
the nodes to be individually reconfigured under the control of a system manager, in order 
to add or remove a node from ring operations. 

The foregoing description has been limited to a specific embodiment of this 
invention. It v^U be apparent, however, that variations and modifications may be made 



8 



H:\104\141\0004\PROSECUT\PATAPP.doc 





PATENT 
104141-0004 



to the invention, such as, for example, use of any type of commimication media between 
nodes, configuration of the ring as any type of dual-path communication network, with 
the attainment of some or all of its advantages. Therefore, it is the object of the appended 
claims to cover all such variations and modifications as come within the true spirit and 
5 scope of the invention. 



What is claimed is: 
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